Residential thermal load components from ResStock #524

jmythms · 2025-09-08T16:38:14Z

This PR will update the current thermal load components data to the latest BuildStock data.

RESSTOCK vs Current Scout Data comparison:

Key notes:

Cooling Infiltration: Major sign differences. And large numerical diffs.
Cooling Equipment: High value for Resstock (diff is ~40%). Points towards other large components causing Equip to balance out the normalized value.
Pacific Multi Family and Mobile Home shows large discrepancies across all components.
New England mobile home show large Equip and Infil discrepancies.
Heating matches better than cooling; differences are focused in Wind_cond in mobile homes.

How to interpret these plots:

Color shows heating vs cooling.
Sign change happens in quadrants II and IV.
A large distance from the diagonal line means there was a large difference between new and old data.
Components are normalized values and can be negative or <1.

More stats:

HEATING

Component	Mean diff	MAE	Sign agree (%)	Corr
EQUIP	0.162	0.164	85.185	0.904
GRND	0.021	0.108	100.000	0.189
INFIL	0.035	0.080	100.000	0.703
ROOF	-0.094	0.099	100.000	0.499
WALL	0.017	0.067	100.000	0.621
WIND_COND	-0.172	0.172	100.000	0.499
WIND_SOL	-0.013	0.034	100.000	0.787

COOLING

Component	Mean diff	MAE	Sign agree (%)	Corr
EQUIP	0.401	0.401	100.000	0.152
GRND	0.027	0.091	70.370	0.216
INFIL	-0.220	0.220	18.519	0.707
ROOF	-0.084	0.086	100.000	0.279
WALL	-0.011	0.078	92.593	-0.021
WIND_COND	-0.060	0.063	51.852	0.626
WIND_SOL	0.011	0.098	100.000	0.558

Calculation methods:
Mean diff = (ResStock-ScoutCurrentData).mean()
MAE = (|ResStock-ScoutCurrentData|).mean()
Sign Agree = (if sign_ResStock == sign_ScoutCurrentData).mean() to percentage
Correlation = ResStock.corr(ScoutCurrentData)

jmythms · 2025-09-08T16:41:19Z

Questions:

Are we currently using the Climate Zone conversions in the R file in later Scout processing?
There are no specific "people" components in the RESSTOCK file. There is however an internal_gains component? Repurpose that somehow?
1121 Buildings with nan type (out of 550,000 buildings).

jmythms · 2025-09-08T16:42:39Z

scout/supporting_data/thermal_loads_data/resstock_to_thermalLoads.py

+RESSTOCK_SEGMENT_TO_CATEGORY = {
+    # Windows / skylights
+    "windows_conduction": "WIND_COND",
+    "skylights_conduction": "WIND_COND",
+    "windows_solar": "WIND_SOL",
+    "skylights_solar": "WIND_SOL",
+    # Roof / ceilings
+    "roofs": "ROOF",
+    "ceilings": "ROOF",
+    # Walls / opaque
+    "walls": "WALL",
+    "foundation_walls": "WALL",
+    "rim_joists": "WALL",
+    "doors": "WALL",
+    # Infiltration / ventilation
+    "infiltration": "INFIL",
+    "mechanical_ventilation": "INFIL",
+    "natural_ventilation": "INFIL",
+    # Ground contact
+    "slabs": "GRND",
+    "floors": "GRND",
+    # Internal/equipment gains
+    "internal_gains": "EQUIP",
+    "internal_mass": "EQUIP",
+    "lighting": "EQUIP",
+    "whole_house_fan": "EQUIP",
+    "ducts": "EQUIP",
+}


Components: Resstock to Scout

jmythms · 2025-09-08T16:43:47Z

scout/supporting_data/thermal_loads_data/resstock_to_thermalLoads.py

+BLDG_MAPPING = {
+   '50 or more Unit': 'multi family home',
+   'Single-Family Detached': 'single family home',
+   'Single-Family Attached': 'single family home',
+   'Mobile Home': 'mobile home',
+   '20 to 49 Unit': 'multi family home',
+   '5 to 9 Unit': 'multi family home',
+   '3 or 4 Unit': 'multi family home',
+   '2 Unit': 'multi family home',
+   '10 to 19 Unit': 'multi family home',
+   'nan': None
+}


Building types mapping: RESSTOCK to Scout

jmythms · 2025-09-08T17:23:18Z

Questions:

Are we currently using the Climate Zone conversions in the R file in later Scout processing?

There are no specific "people" components in the RESSTOCK file. There is however an internal_gains component? Repurpose that somehow?

1121 Buildings with nan type (out of 550,000 buildings).

@jmythms to lump components in ecm_prep.py, mix in people component into internal gains.

jmythms · 2025-09-24T17:17:13Z

scout/supporting_data/thermal_loads_data/Com_TLoads_Process.R

+# ==============================================================================
+# COMMERCIAL THERMAL LOADS DATA PROCESSING SCRIPT
+# ==============================================================================
+# This script processes base thermal loads data for commercial buildings and 
+# converts it from climate zone-based to census division-based organization
+# for use in building energy modeling and analysis.
+#
+# Input: Com_TLoads_Base.csv - Base thermal loads data by climate zone
+# Output: Com_TLoads_PrepFinal.txt - Processed thermal loads by census division
+#         Com_Cdiv_Czone_ConvertTablePrep.txt - Conversion factors
+#         Com_Cdiv_Czone_ConvertTablePrep_Rev.txt - Reverse conversion factors
+# ==============================================================================


Should we keep the R scripts here?

Lets delete them

jmythms · 2025-09-24T17:17:35Z

scout/supporting_data/thermal_loads_data/Res_TLoads_Process.R

+# ===============================================================================
+# Residential Thermal Loads Processing Script
+# ===============================================================================
+# This script processes residential thermal loads data and creates conversion 
+# tables for mapping between census divisions and climate zones.
+#
+# Part 1: Process residential thermal loads by building type and end use
+# Part 2: Create census division to climate zone conversion tables from RECS data
+# ===============================================================================


Residential scripts are here.

jmythms · 2025-09-24T18:09:18Z

scout/supporting_data/thermal_loads_data/resstock_to_thermalLoads.py

+ALL_COLS = [
+    "Unnamed: 0",
+    "building_id",
+    "job_id",
+    "started_at",
+    "completed_at",


Remove this

jmythms · 2025-09-24T18:16:18Z

This PR will update the current thermal load components data to the latest BuildStock data.

RESSTOCK vs old data comparison:

Component Mean diff MAE Sign agree (%) Corr
EQUIP 0.281 0.283 92.593 0.908
GRND 0.024 0.100 85.185 0.697
INFIL -0.093 0.150 59.259 0.921
ROOF -0.089 0.092 100.000 0.502
WALL 0.003 0.072 96.296 0.702
WIND_COND -0.116 0.118 75.926 0.902
WIND_SOL -0.001 0.066 100.000 0.945

Divide heating and cooling

jmythms · 2025-09-24T18:23:22Z

verify if the other mseg_* files changed or not?

…k/scout into issue-486/thermal_loads

aspeake

The core script to generate Scout-compatible files looks to be correct, and I was able to successfully run it myself. I have a few questions, but I don't think my changes will result in new input files (or at least not change the final results).

Regarding the comparisons, I think they look pretty reasonable. The components/end uses that jump out tend to hover around zero so the relative differences can be high, or they are related to mobile homes which have a low saturation, and I trust their representation ResStock. That is to say that I think the differences are due to modeling assumptions, and not the methods implemented in this PR.

aspeake · 2025-10-20T21:28:47Z

docs/thermal_load_components.rst

-
+    * Map residential table building types to AEO building types.
+        * Single-Family Detached -> single family home
+        * Single-Family Attached -> single family home


@jtlangevin does this look right or should SFA map to multi family?

aspeake · 2025-10-20T21:29:21Z

docs/thermal_load_components.rst

 * Manually input data from the PDF publications, as in :repo_file:`Res_TLoads_Base.csv` and :repo_file:`Com_TLoads_Base.csv`.

-3. Condense CSV tables into final set of thermal load components data using an R script.
+3. Condense CSV tables into final set of thermal load components data using an R/Python script.


Suggested change

3. Condense CSV tables into final set of thermal load components data using an R/Python script.

3. Condense CSV tables into final set of thermal load components data.

aspeake · 2025-10-20T21:32:46Z

scout/ecm_prep.py

+    Non-destructive: original component nodes are preserved under
+    ['internal gains']['components_original'].


aspeake · 2025-10-20T21:38:13Z

scout/ecm_prep.py

+            for fuel_key, fuel_val in bldg_val.items():
+                if not isinstance(fuel_val, dict):
+                    continue
+                # print(f"Processing {geo_key} {bldg_key} {fuel_key}")


delete comment

aspeake · 2025-10-20T21:38:51Z

scout/supporting_data/thermal_loads_data/Com_TLoads_Process.R

+# ==============================================================================
+# COMMERCIAL THERMAL LOADS DATA PROCESSING SCRIPT
+# ==============================================================================
+# This script processes base thermal loads data for commercial buildings and 
+# converts it from climate zone-based to census division-based organization
+# for use in building energy modeling and analysis.
+#
+# Input: Com_TLoads_Base.csv - Base thermal loads data by climate zone
+# Output: Com_TLoads_PrepFinal.txt - Processed thermal loads by census division
+#         Com_Cdiv_Czone_ConvertTablePrep.txt - Conversion factors
+#         Com_Cdiv_Czone_ConvertTablePrep_Rev.txt - Reverse conversion factors
+# ==============================================================================


Lets delete them

aspeake · 2025-10-21T15:32:29Z

scout/supporting_data/thermal_loads_data/resstock_to_thermalLoads.py

I am less picky on documentation and design of these scripts, but this function could use a more detailed description.

aspeake · 2025-10-21T15:42:38Z

scout/supporting_data/thermal_loads_data/resstock_to_thermalLoads.py

Should be able to just do an average instead of a weighted average

scout/supporting_data/thermal_loads_data/resstock_to_thermalLoads.py

aspeake · 2025-10-21T16:15:17Z

scout/supporting_data/thermal_loads_data/Res_TLoads_Process.py

I'm confused what this script does? Is it redundant with resstock_to_thermalLoads.py?

It is a python equivalent to scout\supporting_data\thermal_loads_data\Res_TLoads_Process.R. If we are not keeping the R scripts, delete this as well?

What does it do? Does it do something different than resstock_to_thermalLoads.py?

It does the equivalent of resstock_to_thermalLoads.py, but for the previous original dataset using RECS microdata, etc.

Ah okay, in that case I think we can delete it

aspeake · 2025-10-21T16:40:09Z

scout/supporting_data/stock_energy_tech_data/mseg_res_com_cz.json

Are all the mseg files generated using the CDIV component load file? And just mapped in teh translator script?

jmythms self-assigned this Sep 8, 2025

jmythms commented Sep 8, 2025

View reviewed changes

jmythms marked this pull request as ready for review September 18, 2025 20:53

jmythms requested a review from aspeake September 18, 2025 21:03

jmythms marked this pull request as draft September 24, 2025 15:02

jmythms commented Sep 24, 2025

View reviewed changes

jmythms marked this pull request as ready for review October 2, 2025 15:31

jmythms added 12 commits October 7, 2025 10:53

working r scripts

92e3a1f

R to Python translation

647c020

Resstock code + output file

9d51262

Updated thermal loads file + comversion file

6cc1f6a

Updated mseg_res_com_cz.json for new thermal loads

48f604d

formatting changes

50ac394

format fix 2

e7a2f7d

fix test

1bb8391

reorder column names and round decimal to 4

9dfdeb3

delete temp output file

abbee25

update other mseg_ files

263e9d3

update column names, better function names

5f6d082

jmythms force-pushed the issue-486/thermal_loads branch from fe7a4b5 to 5f6d082 Compare October 7, 2025 15:26

jmythms and others added 3 commits October 7, 2025 11:40

fix flake8 errors

840766a

Upload results files from CI build

10b97ab

Update documentation

38ab2c2

jmythms added 2 commits October 9, 2025 11:37

Merge branch 'issue-486/thermal_loads' of https://github.com/trynthin…

b3bc667

…k/scout into issue-486/thermal_loads

remove unused all columns list

dc08aa1

jtlangevin linked an issue Oct 14, 2025 that may be closed by this pull request

Update thermal load components data #486

Open

jmythms requested a review from jtlangevin October 14, 2025 17:41

jmythms changed the title ~~Thermal load components from BuildStock~~ Residential thermal load components from ResStock Oct 14, 2025

jtlangevin added this to the v1.2.0 milestone Oct 14, 2025

aspeake requested changes Oct 21, 2025

View reviewed changes

quick review comment fixes

663e527

	3. Condense CSV tables into final set of thermal load components data using an R/Python script.
	3. Condense CSV tables into final set of thermal load components data.

		Non-destructive: original component nodes are preserved under
		['internal gains']['components_original'].

Residential thermal load components from ResStock #524

Are you sure you want to change the base?

Residential thermal load components from ResStock #524

Uh oh!

Conversation

jmythms commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmythms commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmythms commented Sep 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmythms commented Sep 24, 2025

Uh oh!

jmythms commented Sep 24, 2025

Uh oh!

aspeake left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aspeake Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmythms Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jmythms commented Sep 8, 2025 •

edited

Loading

jmythms commented Sep 8, 2025 •

edited

Loading

aspeake Oct 20, 2025 •

edited

Loading

jmythms Oct 23, 2025 •

edited

Loading